Skip to content

Comments

init compressDrv and compressDrvWeb#292324

Merged
Ma27 merged 2 commits intoNixOS:masterfrom
motiejus:compress-drv
Aug 4, 2024
Merged

init compressDrv and compressDrvWeb#292324
Ma27 merged 2 commits intoNixOS:masterfrom
motiejus:compress-drv

Conversation

@motiejus
Copy link
Contributor

Description of changes

add compressDrv and compressDrvWeb. I have been working on adding gamja with passthru.data-compressed and @K900 suggested to add something generic. This is my first attempt.

Things done

compressDrv compresses files in a given derivation.

compressDrvWeb compresses a derivation for a loosely-defined
pre-compressed "web server" usage.

This intends to replace the passthru.data-compressed derivations that have accumulated in nixpkgs with something more reusable. First one is gitea.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@motiejus
Copy link
Contributor Author

@ofborg build tests.compress-drv

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-linux: 1 This PR causes 1 package to rebuild on Linux. labels Feb 29, 2024
@motiejus motiejus marked this pull request as ready for review February 29, 2024 13:33
@motiejus
Copy link
Contributor Author

motiejus commented Feb 29, 2024

adding @K900 who suggested the original idea

cc @disassembler @kolaente @Ma27 @techknowlogick gitea maintainers.

@philiptaron
Copy link
Contributor

Mind fixing the merge conflicts? I'll take a closer look after that.

@motiejus
Copy link
Contributor Author

Sure, I'll do it within the next couple of days.

@motiejus
Copy link
Contributor Author

Mind fixing the merge conflicts? I'll take a closer look after that.

Rebased.

Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several non-blocking comments.

@techknowlogick
Copy link
Member

Thanks @motiejus! My apologies for submitting a PR to the Gitea derivation earlier that conflicted this PR.

I'm not familiar enough with this area of change to be able to offer any other feedback than "Looks good to me, and works for Gitea". From my limited perspective this appears to be a good generalization, and being able to utilize it for other derivations is definitely a big positive.

@motiejus
Copy link
Contributor Author

motiejus commented Jul 1, 2024

Thank you for your comments. I've addressed all of them in my private branch, but discovered that the test is flaky. Turns out process substitution with >( ... ) is asynchronous -- bash doesn't wait for those substituted processes and can terminate early, causing the application to fail.

I've been trying different approaches without process substitution, but didn't get there yet. Will re-request your review with v2.

@philiptaron
Copy link
Contributor

Mind marking this as draft while working on that, @motiejus? I'll be happy to review when you're ready.

@motiejus motiejus marked this pull request as draft July 4, 2024 18:09
@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation This PR adds or changes documentation 8.has: changelog This PR adds or changes release notes labels Jul 5, 2024
@motiejus motiejus force-pushed the compress-drv branch 3 times, most recently from 3bd8637 to 9f3dec4 Compare July 5, 2024 20:29
@motiejus
Copy link
Contributor Author

motiejus commented Jul 9, 2024

There was something off with sem --id. If anyone reading this is curious, run both commands and compare outputs:

# this is good
while read -r i; do sem  -j 50 "sleep 1; echo $i"; done < <(seq 100)
# this is bad
while read -r i; do sem --id x  -j 50 "sleep 1; echo $i"; done < <(seq 100)

I opted for a simple, though not 100% efficient approach (may come back to it later). This seems to work with everything.

@motiejus motiejus closed this Jul 9, 2024
@motiejus motiejus reopened this Jul 9, 2024
@motiejus motiejus marked this pull request as ready for review July 9, 2024 04:50
Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything checks out. It's a nice and simple PR now.

Comment on lines 62 to 63
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love to see lndir gone here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why is lndir that bad?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not bad per se -- it's just part of the the xorg world, and replacing it with something that's natively in the stdenvNoCC is both simpler and requires less learning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually like lndir more, because it's quite widely used, a single operation (instead of two traversals) and automatically resolves nested symlinks (which the previous PR version didn't). I just couldn't get it to work before, and no longer understand/remember why.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not aware of any tool replacing lndir

@philiptaron philiptaron added the 12.approvals: 1 This PR was reviewed and approved by one person. label Jul 13, 2024
Comment on lines 62 to 63
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why is lndir that bad?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My knowledge on the documentation infrastructure is a little outdated, so excuse my ignorance, but does this doc-comment appear in one of the manuals?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how to check; I mostly cargo-culted the comment format from the functions I like in lib/.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @NixOS/documentation-team any suggestions?

Copy link
Contributor

@hsjobeki hsjobeki Aug 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using doc-comments would be beneficial, since they can be retrieved with :doc in the repl. since release 2.24
And we plan to render them into the manual.

/** */

Content should be markdown, so its rendered properly

My knowledge on the documentation infrastructure is a little outdated, so excuse my ignorance, but does this doc-comment appear in one of the manuals?

No not yet.

@motiejus It is planned to migrate all doc comments into one common format:

https://github.com/NixOS/rfcs/blob/c65c8321782b40844167beb48818471f70900d9d/rfcs/0145-doc-strings.md#migration-example

(:doc doesn't work in this particular case, because some features are still missing, but https://noogle.dev should find it after its merged)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did my best following the examples here and in attrsets.

My system is still on 2.18, which makes it nontrivial to see the end result. I'll have a look once this lands somewhere or my nix is 2.24.

@motiejus motiejus force-pushed the compress-drv branch 3 times, most recently from 1da5bbe to 1e6c24b Compare July 29, 2024 12:28
@motiejus motiejus requested a review from Ma27 July 29, 2024 12:29
Copy link
Member

@Ma27 Ma27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

*compressDrv* compresses files in a given derivation.

*compressDrvWeb* compresses a derivation for a loosely-defined
pre-compressed "web server" usage.

This intends to replace the `passthru.data-compressed` derivations that
have accumulated in nixpkgs with something more reusable.
@Ma27 Ma27 merged commit bddcfad into NixOS:master Aug 4, 2024
@motiejus motiejus deleted the compress-drv branch August 5, 2024 06:45
@hsjobeki
Copy link
Contributor

hsjobeki commented Aug 5, 2024

Fiy: https://noogle.dev/f/pkgs/compressDrv

@philiptaron
Copy link
Contributor

Looks like it could use a little markdown editing to look a bit better, but it's working!

@hsjobeki
Copy link
Contributor

hsjobeki commented Aug 5, 2024

Looks like it could use a little markdown editing to look a bit better, but it's working!

I've done that If you want to merge it

#332466

motiejus added a commit to motiejus/nixpkgs that referenced this pull request Aug 5, 2024
This is a leftover from the original versions of NixOS#292324. Reduces the
API surface to not accept accidental arguments.
@SuperSandro2000
Copy link
Member

@Ma27 to be able to use this for nextcloud we also need to pass -not -iregex ".*(\/apps\/.*\/l10n\/).*". Maybe I come up with something.

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Aug 6, 2024

I am currently doing a PR with some small improvements including zstd support which chrome is already using.

PS Coincidentally this broke eval because I used that name locally already 😂

Edit: #332752

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog This PR adds or changes release notes 8.has: documentation This PR adds or changes documentation 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants